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FOREWORD 


In  June  1995,  Army  Training  XXI  was  formally  acknowledged  as  the  training  component 
for  Force  XXI,  the  Army’s  redesigned  force  of  the  next  centuiy'.  Army  Training  XXI  consists  of 
three  axes:  Warfighter  XXI,  Warrior  XXI,  and  the  Warrior  Network  (WARNET  XXI).  The 
objective  of  Warfighter  XXI  and  Warrior  XXI  is  to  help  develop  or  reengineer  the  three  pillars  of 
Army  training;  institutional,  self-development,  and  unit  training  with  a  specific  focus  on  digitized 
training.  WARNET  XXI  provides  for  the  linkage  of  training  acquisition,  new  equipment  training, 
and  digitization  of  systems  training  support  products. 

Concurrent  with  these  efforts  was  the  implementation  of  the  WARNET  Pilot  initiative. 

The  role  of  WARNET  Pilot  is  to  synchronize  the  development  and  delivery  of  new  technologies 
in  training.  More  specifically,  the  intent  is  to  use  distance  learning  and  information  technologies  to 
pilot  the  delivery  of  high-quality,  standardized  training  to  soldiers  and  units  where  and  when  they 
need  it.  The  WARNET  Pilot  Team  was  formed  at  the  U.S.  Army  Infantry  Center,  under  the 
Directorate  of  Operations  and  Training  (DOT)  -  U.S.  Army  Infantry  School  (USAIS)  with  the 
support  of  the  National  Guard  Bureau  to  manage  multimedia  distance  learning  and  distributive 
training  development  for  USAIS.  In  concert  with  other  training  activities  at  USAIS,  the 
WARNET  Pilot  Team  was  given  the  responsibility  for  developing  Infantry-specific  courses. 

In  November  1995,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  Infantry  Forces  Research  Unit  (IFRU)  was  contacted  by  the  Office  of  the  Special 
Assistant  to  the  Commanding  General,  Army  National  Guard  to  provide  technical  assistance  for 
the  WARNET  Pilot  Team  course  development  effort.  This  report  describes  IFRU’s  work  in  the 
formative  evaluation  of  the  computer-based  Tactics  Certification  Course  (TCC)  -  Principles  of 
War  Module.  The  TCC  is  the  first  core  Infantry  resident  course  to  undergo  the  conversion  from  a 
resident  classroom/lecture-based  format  to  a  distance  learning  format.  This  report  provides  a 
model  for  evaluating  the  effectiveness  of  newly  developed  courses  with  multimedia  formats.  In  so 
doing,  it  provides  the  means  for  ensuring  that  the  distance  learning  technologies  selected  for  the 
TCC  are  optimally  employed.  The  results  of  this  research  were  briefed  to  course  developers,  the 
WARNET  Pilot  Team,  and  USAIS-DOT  staff. 


ZITA  M.  SIMUTIS  EDGAR  M.  JOHNSON 

Technical  Director  Director 
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PRELIMINARY  EVALUATION  OF  THE  COMPUTER-BASED  TACTICS  CERTIFICATION 
COURSE-PRINCIPLES  OF  WAR  MODULE 

EXECUTIVE  SUMMARY 


Research  Requirements: 

In  1995  the  U.S.  Army  Infantry  School  (USAJS)  at  Fort  Benning  was  selected  by  the 
Deputy  Chief  of  Staff  for  Training  as  the  lead  agency  for  implementing  the  WARNET  Pilot 
initiative.  The  role  of  the  WARNET  Pilot  is  to  use  distance  learning  and  information  technologies 
to  pilot  the  delivery  of  high-quality,  standardized  training  to  soldiers  and  units  where  and  when  they 
need  it.  The  WARNET  Pilot  Team  was  given  the  responsibility  for  developing  selected  Infantiy-- 
specific  courses.  The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  Infantry' 
Forces  Research  Unit  at  Fort  Benning  agreed  to  provide  technical  assistance  for  the  course 
development  effort.  This  research  describes  a  portion  of  the  formative  evaluative  process  used  in 
developing  the  computer-based  Principles  of  War  module  for  the  Tactics  Certification  Course. 

Procedure: 

Sixteen  subjects  from  USAIS  participated  in  the  research.  Subjects  were  randomly  assigned 
to  one  of  two  groups.  The  experimental  group  received  the  computer-based  instructional  version  of 
the  Principles  of  War  module  and  an  end-of-module  quiz.  Subjects  assigned  to  the  control 
condition  received  only  the  end-of-module  quiz.  In  addition  to  the  quiz,  subjects  completed  a 
background/computer  experience  survey  and  a  questionnaire  assessing  their  opinions  on  selected 
aspects  of  the  Principles  of  War  module. 

Findings. 

Subjects  in  the  experimental  group  answered  significantly  more  quiz  items  correctly 
(88.9%)  than  did  subjects  in  the  control  condition  (48.1%).  Ratings  of  selected  aspects  of  the 
module  varied.  Certain  sections  of  the  module  clearly  needed  to  be  modified.  Other  sections 
required  only  minor  refinements.  The  overall  ratings  of  the  instructional  value  of  the  course  were 
positive. 

Utilization  of  Findings: 

The  findings  from  this  research  were  briefed  to  course  developers,  the  WARNET  Pilot 
Team,  and  USAIS  staff.  The  results  of  this  feedback  will  be  used  to  refine  selected  areas  of  the 
module.  The  formative  evaluation  process  followed  in  this  research,  and  subsequent  TCC  modules 
will  help  ensure  that  the  distance  learning  technologies  selected  for  the  TCC  are  optimally 
employed. 
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PRELIMINARY  EVALUATION  OF  THE  COMPUTER-BASED  TACTICS 
CERTIFICATION  COURSE  -  PRINCIPLES  OF  WAR  MODULE 


Introduction 

In  December  1995  the  TRADOC  Deputy  Chief  of  Staff  for  Training  (DCST) 
selected  the  U.S.  Army  Infantry  School  (USAIS)  at  Fort  Benning,  Georgia  as  the  lead 
agency  for  implementing  the  WARNET  Pilot  initiative.  The  role  of  WARNET  Pilot  is 
to  synchronize  the  development  and  delivery  of  new  technologies  in  training.  More 
specifically,  the  intent  is  to  use  distance  learning  and  information  technologies  to  pilot  the 
delivery  of  high  quality,  standardized  training  to  soldiers  and  units  where  and  when  they 
need  it. 

WARNET  Pilot  Team 

The  WARNET  pilot  team  was  formed  at  the  United  States  Army  Infantry  Center 
(USAIC)  under  the  Directorate  of  Training  (DOT)  -  USAIS,  with  the  support  of  the 
National  Guard  Bureau  (NGB)  to  manage  multimedia  distance  learning  (MMDL)  and 
distributive  training  development  for  USAIS.  Guidance  was  given  to  the  Infantry  School 
for  development  of  Total  Army  Training  System  (TATS)  courseware  under  the  Total 
Army  School  System  (TASS)  initiative,  which  was  directed  by  TRADOC.  The 
WARNET  pilot  team,  in  conjunction  with  the  Training  Support  Center  (TSC)  under  DOT 
was  given  the  responsibility  for  developing  Infantry-specific  courses.  This  initiative  was 
based  on  specific  guidance  from  HQ,  TRADOC  TASS  OPLAN-2,  effective  1  January 
1995  (WARNET  Pilot  Team  TATSC  Management  Plan,  February,  1996). 

WARNET  Pilot  Approach  to  Course  Development 

The  WARNET  pilot  team  and  DOT-USAIS  have  initiated  planning  in  the 
following  areas:  multimedia  distance  learning,  distributive  training,  and  interactive 
courseware.  As  currently  envisioned,  separate  active  component  (AC)  and  reserve 
component  (RC)  courseware  will  be  phased  out.  In  its  place  only  Total  Army  Training 
System  Courses  (TATSC)  will  remain  for  seamless  training  of  AC  and  RC  soldiers.  To 
insure  seamless  training,  TATSC  will  incorporate  various  ‘state  of  the  art  technologies’, 
such  as  computer-based  instruction  (CBI)  and  compact  disk  read-only-memory 
(CD-ROM).  These  technologies  provide  the  necessary  flexibility  to  accommodate  both 
AC  and  RC  training  needs,  scheduling  demands,  and  logistics  issues  not  possible  through 
approaches  currently  used,  e.g.,  live  classroom  lecture  (WARNET  Pilot  Team,  1996). 

The  WARNET  pilot  team  will  develop  and  coordinate  the  conversion  of  RC 
courses  to  TATS  courseware  to  support  the  TASS  objective  of  establishing  decentralized 
training  at  multiple  “teaching  sites”  available  to  all  soldiers.  The  WARNET  pilot 
approach  to  course  development  involves,  in  part,  the  identification  of  selected  infantry 
courses  for  reconfiguration;  development  of  a  prototype  course;  course/courseware 
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evaluation,  testing  and  refinement,  followed  by  a  more  ambitious  phase  entailing 
simultaneous  development  and  refinement  of  new  courses.  The  focus  of  the  WARNET 
pilot  effort  is  directed  toward  the  production  and  development  of  five  short  courses: 
Tactics  Certification  Course  (TCC),  Teach,  Assess,  Coimsel  -  Training  and  Orientation 
Course,  Instructor-Trainer  Course,  Infantry  Mortar  Leader  Course,  and  the  Bradley 
Fighting  Vehicle  Leaders  Course  (WARNET  Pilot  Team  TATSC  Management  Plan, 
February,  1996). 

The  TCC  was  selected  as  the  prototype  course  based  on  two  criteria.  The  first 
was  that  all  Officer  Candidate  School  (OCS)  and  Career  Management  Field  (CMF)  1 1 
course  instructors  are  required  to  be  TCC  qualified.  In  order  to  establish  the  base  of 
instructors  for  the  implementation  of  TASS  in  the  seven  regions  designated  by  TRADOC 
located  throughout  the  U.S.,  TCC  would,  logically,  have  to  be  the  first  course.  The 
second  rationale  for  selecting  the  TCC  was  based  on  time  and  experience.  A  reasonably 
short  course  had  to  be  selected  to  prove  the  technology  and  concept  would  work.  This 
would  give  USAIS  the  data  base  and  experience  upon  which  to  structure  future  course 
development.  The  TCC  was  selected  because  it  has  a  10  day  resident  phase  Program  of 
Instruction  (POI)  that  could  be  reduced  to  a  shorter  resident  phase  and  a  nonresident 
phase  using  distance  learning  technology  such  as  CD-ROM  and  CBI.  The  overall 
concept  for  TATSC  development  is  to  create  courseware  that  uses  technology  to  gain 
maximum  learning  for  the  student  with  a  minimum  of  resident  school  attendance 
(WARNET  Pilot  Team  TATSC  Management  Plan,  February,  1996). 

ARI  ’s  Involvement  with  the  WARNET  Pilot  Team 


In  November  1995,  the  ARI-Infantry  Forces  Research  Unit  (IFRU)  was  contacted 
by  the  Office  of  the  Special  Assistant  to  the  Commanding  General,  Army  National  Guard 
to  provide  technical  assistance  for  the  WARNET  Pilot  Team  course  development  effort. 
Specifically,  ARI-  IFRU’s  assistance  was  requested  in  the  following  areas:  identification 
and  selection  of  an  appropriate  instructional  model  (or  models)  to  guide  course  (TCC) 
development;  review  of  new  course  structure  and  format  to  ensure  WARNET  Pilot  Team 
course  developers  adhered  to  the  major  tenets  of  the  instructional  model(s);  development 
of  an  evaluation  plan  to  compare  the  effectiveness  of  the  new  course  vs  traditional 
method(s)  of  instruction;  assistance  in  course  evaluation;  computation  of  all  statistical 
analyses  and  delivery  of  feedback  from  the  evaluation  to  course  developers,  WARNET 
Pilot  Team,  and  USAIS-DOT  staff. 

TCC:  Course  Organization 

In  February  and  March  1996,  two  researchers  from  ARI -IFRU  monitored  parts  of 
two  separate  TCC  classes.  Classes  consist  of  six  to  eleven  students  with  one  instructor. 
The  TCC,  as  presently  configured,  involves  four  days  of  lecture  and  small  group 
instruction,  a  one  day  tactical  exercise  without  troops  (TEWT),  followed  by  a  two  hour 
end-of-course  exam  on  the  sixth  day.  Content  areas  include  Army  operations, 
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operational  symbols,  troop  leading  procedures,  offensive  operations,  defensive 
operations,  and  a  brief  overview  on  engineer  and  fire  support  operations. 

Students  are  provided  with  various  handouts  and  reading  materials,  primarily 
chapters  taken  from  Army  field  manuals  (FMs).  An  open  book,  short  answer/essay  “pre¬ 
test”  is  given  to  the  students  as  a  take  home  assignment  prior  to  the  start  of  the  course. 
With  regard  to  course  evaluation,  three  20  minute  quizzes  are  given  at  the  beginning  of 
each  class  covering  key  points  from  the  previous  day’s  instruction.  Quizzes  are  primarily 
short  answer/essay.  In  addition,  a  one  hour  operational  symbols  quiz  is  administered 
during  the  second  day.  The  quiz  consists  of  a  mission  scenario  in  which  students  are 
asked  to  use  the  symbols  to  designate  units  and  various  actions  on  a  map  overlay. 
Feedback  is  provided  relatively  quickly  following  each  quiz.  The  final  exam  is,  again, 
primarily  short  answer/essay. 

Significant  hands-on  time  is  devoted  to  practical  exercises  in  the  development  and 
presentation  of  operational  orders  (OPORDs).  Additional  time  is  allotted  for  constructing 
map  overlays.  Students  are  encouraged  to  become  actively  involved  in  the  topic 
discussions.  Large  parts  of  the  engineer  and  fire  support  sections  are  presented  on  video¬ 
tape  depicting  these  particular  combat  support  units  in  action.  This  is  accompanied  by  a 
general  overview  of  these  units’  functions  and  the  equipment  that  they  use.  The  TEWT 
entails  having  students  walk  the  terrain  representing  the  actual  site  on  which  the  student’s 
OPORD  was  based.  Some  students  also  brief  their  OPORD  if  they  have  not  already  done 
this  during  the  classroom  practical  OPORD  exercise. 

TCC:  Course  Redesign  and  Development 

Based  in  part  on  ARI  researchers’  analysis  of  course  objectives  and  content,  it 
appeared  that  much  of  the  TCC  small  group  instruction  could  be  eliminated  and  replaced 
with  a  more  efficient  CD-ROM  computer  based  instructional  format. 

A  literature  review  was  conducted  to  identify  candidate  instructional  models  to 
guide  the  development  of  the  new  course.  A  key  objective  of  the  WARNET  Pilot  Team 
was  to  successfully  develop  an  instructional  approach  that  ensured  learning  and  high 
levels  of  retention.  With  this  in  mind,  the  instructional  model  selected  was  based,  in 
large  part,  on  the  Personalized  System  of  Instruction  (PSI)  [Buskist,  Cush,  & 

DeGrandpre,  1991].  PSI  is  characterized  by  the  following  qualities:  self-pacing,  unit 
mastery,  lectures  and  demonstrations,  written  communication,  and  the  use  of  proctors. 

For  a  complete  description  of  the  PSI  approach  and  supporting  empirical  documentation, 
see  Appendix  A. 

Revised  PSI  Model  as  Applied  to  the  TCC 

The  modified  version  of  the  PSI  model  proposed  for  the  computer-based/CD- 
ROM  section  of  the  TCC  has  several  key  features.  First,  students  must  demonstrate  high 
levels  of  mastery  of  unit  (module)  content  before  they  can  move  on  to  the  next  unit. 
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Unit  (module)  quizzes  are  predominantly  matching  (The  quiz  for  the  prototype  module 
which  was  used  in  this  evaluation  was  multiple-choice.)  For  the  TCC,  unit  mastery 
requires  that  the  student  answer  all  doctrinal  definitions  correctly.  The  exact  number  of 
items  and  percentage  required  for  mastery  will  vary,  however,  depending  on  the  types  of 
questions,  i.e.,  definition,  knowledge,  application,  answered  by  the  student. 

Second,  students  receive  immediate  performance  feedback  consisting  of  a  total 
correct  score  (and  percentage  correct)  and  a  score  for  each  of  the  content  areas  for  the 
module  (a  total  correct  score  and  specific  item  feedback  for  each  principle  of  war).  Those 
students  whose  scores  fail  to  meet  the  selected  criteria  are  directed  back  to  the  section(s) 
in  the  lesson(s)  where  they  had  missed  items  and  allowed  to  review  relevant  course 
material.  Following  the  review,  students  are  quizzed  (using  a  multiple-choice  format)  on 
the  appropriate  section(s).  Once  the  student  has  satisfactorily  demonstrated  mastery  of 
the  sections  or  content  areas  in  question,  he  or  she  is  allowed  to  move  on  to  the  next  unit 
or  module. 

Third,  interspersed  within  the  unit  are  summary'  (self-assessment)  questions 
focusing  the  student’s  attention  on  key  points  to  be  learned  for  each  lesson. 

Feedback  (immediately)  following  the  completion  of  the  self-assessment  questions 
ensures  that  the  student  stays  on  track  and  is  actively  involved  in  the  instructional 
process.  For  the  present  module  (Principles  of  War),  three  multiple-choice  questions  are 
included  at  the  end  of  the  review  section  for  each  lesson.  Students  must  answer  all  items 
correctly  before  they  can  move  on  to  the  next  lesson  within  the  module. 

No  set  deadlines  for  completing  each  section  or  quiz  are  provided,  although  a 
general  course  deadline  is  established.  For  the  most  part,  students  can  proceed  at  their 
own  pace. 

Consistent  with  the  self-pacing  characteristic  of  PSI,  an  option  is  provided  for  the 
student  to  test  out  of  selected  lessons  or  the  entire  module.  The  test  is  identical  to  the 
final  quiz  described  earlier  which  involves  matching  statements  to  the  nine  principles  of 
war.  All  doctrinal  definitions  must  be  answered  correctly  or  the  student  must  complete 
the  appropriate  lesson(s)  for  the  principle(s)  of  war  in  question.  For  the  remaining  items, 
which  tap  the  student’s  doctrinal  knowledge  and  ability  to  apply  the  doctrinal  definitions, 
the  criterion  for  mastery  will  vary,  depending  on  the  types  of  questions  correctly 
answered  by  the  student. 

As  was  the  case  for  the  final  quiz,  students  are  provided  with  immediate 
performance  feedback  consisting  of  a  total  correct  score  (and  percentage  correct).  In 
addition,  students  receive  feedback  for  each  area  (principle  of  war)  to  include  a  total  score 
for  the  area  in  question  and  specific  item  feedback. 

Students  who  fail  to  meet  the  criteria  established  for  each  principle  of  war  must 
take  the  lesson(s)  for  the  specific  principle(s)  of  war  in  question.  Once  the  student 
completes  the  assigned  lesson(s),  he  or  she  is  retested  (using  a  multiple-choice  format) 
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only  on  the  selected  lesson(s).  Upon  satisfactory  mastery  of  the  lesson(s),  the  student  is 
allowed  to  move  to  the  next  unit  or  module. 

Development  of  Modules:  Formative  Evaluation 

Using  a  formative  evaluation  plan  adapted  from  Smith  and  Ragan  (1993),  the 
principles  of  war  module  was  carefully  evaluated  at  key  junctures  during  the 
developmental  process.  A  summary  of  the  evaluation  process  is  presented  in  the 
following  sections. 

Goal  review.  The  key  point  at  this  stage  was  to  confirm  that  the  goals  of  the 
reconfigured  TCC  are  representative  of  a  real  instructional  need  and  are  consistent  with 
USAIS/WARNET  Pilot  expectations.  In  a  broad  sense  this  issue  was  confirmed.  One  of 
the  major  objectives  of  the  WARNET  Pilot  initiative  is  to  exploit  distance  learning  and 
information  technologies  to  pilot  the  delivery  of  high  quality,  standardized  training  to 
soldiers  and  units  where  and  when  they  need  it;  thereby  reducing  the  time  and  costs  of 
sending  soldiers  to  the  actual  site  of  the  course  for  instruction.  The  conversion  of  the 
existing  small  group  instruction/lecture-based  TCC  to  a  predominantly  computer-based 
instructional  format  represents  a  major  step  in  addressing  the  instructional  objectives  of 
WARNET  Pilot. 

Expert  reviews.  As  draft  versions  of  the  Principles  of  War  Module  were 
completed,  course  developers,  ARI  researchers,  and  an  instructional  design  expert 
reviewed  course  objectives,  lesson  review  questions,  and  the  end-of-module  quiz  to 
insure  key  objectives  were,  in  fact,  being  assessed.  Checks  were  also  made  to  see  if 
review  (self-assessment)  questions  and  the  end  of  module  quiz  adequately  sampled  the 
domain  that  the  course  objectives  might  cover.  Screen  design  issues  were  also  noted  at 
this  time. 

Before  the  converted  TCC  was  used  by  students,  subject  matter  experts  and  ARI 
researchers  examined  the  content  of  the  instructional  material  for  accuracy  and 
completeness.  Checks  were  made  to  insure  that:  1)  course  content  was  up  to  date;  2) 
examples,  practice  exercises,  and  feedback  were  realistic  and  accurate;  3)  the  instruction 
was  appropriate  for  the  target  learners;  and  4)  the  instructional  strategies  were,  in  fact, 
consistent  with  principles  of  instructional  theory. 

Learner  validation.  During  this  phase,  a  prototype  lesson  (one  of  the  principles  of 
war  -  mass)  was  administered  to  five  Army  non-commissioned  officers  representative  of 
the  target  population.  Questions  addressed  during  this  time  included:  1)  Did  the  students 
understand  the  instruction/textual  material?  2)  Do  the  learners  know  how  to  proceed 
through  the  quizzes,  exams,  practical  exercises?  3)  Can  they  interpret  the  graphics  on  the 
screen? 


During  this  time  the  subjects  were  questioned  about  any  problems  they  were 
having  and  any  misconceptions  the  subjects  might  have  had  were  addressed.  Checks 


were  also  made  to  insure  that  the  language/vocabulary  in  the  instruction  was  familiar  to 
the  subject;  that  the  intent  of  course  objectives/questions  were  clear;  and  that  the  subject 
imderstood  the  explanations/examples  that  were  presented. 

Detailed  notes  from  the  observations  of  the  subjects’  behavior  were  taken. 
Particular  attention  was  paid  to  areas  in  the  instruction  in  which  students  were  unable  to 
proceed;  comments  on  the  lesson  content;  difficulties  in  navigation  through  the  lesson; 
areas  of  instruction  that  were  reviewed  or  revisited;  areas  of  instruction  that  were  ignored; 
and  subjects’  performance  on  the  self-assessment  items.  After  completing  the  lesson,  the 
subjects  were  debriefed  about  their  reactions  to  the  instruction,  specific  computer-based 
instructional  issues,  and  key  points  emerging  from  the  observations.  The  observ'ations 
and  feedback  were  summarized  and  presented  to  the  course  developers  for  use  in  making 
modifications  in  the  instructional  format. 

Small  group  evaluation.  Once  revisions  were  made  to  the  prototype  TCC  lesson 
for  the  principle  of  war  -  mass,  this  corrected  lesson  was  to  be  used  as  a  template  to 
design  lessons  for  the  remaining  eight  principles  of  war.  After  all  lessons  were 
completed,  they  were  then  reviewed  by  ARI  researchers  and  an  instructional  design 
expert.  Feedback  was  summarized  and  presented  to  course  developers. 

The  revised  nine  lesson  TCC  module  was  then  administered  to  a  sample  of 
soldiers  representative  of  the  target  population.  The  objective  of  this  phase  of  the 
evaluation  process  was  to  determine  if  the  selected  instructional  approach  (modified  PSI 
using  a  CBI/CD-ROM  format)  resulted  in  significantly  better  performance  on  the  end-of- 
module  quiz  than  no  instruction  at  all.  In  addition,  researchers  were  also  interested  in 
obtaining  the  subjects’  perceptions  of  the  instructional  approach  employed,  course 
content,  and  the  instructional  value  of  the  Principles  of  War  module.  The  results  of  this 
small  group  evaluation  are  reported  below. 

Method 


Subjects 

Sixteen  subjects  were  recruited  from  the  United  States  Army  Infantry  School  to 
participate  in  this  phase  of  the  research.  Eleven  subjects  were  active  duty  non¬ 
commissioned  officers.  Five  subjects  were  Department  of  the  Army  civilians.  The 
average  age  of  the  subjects  was  39.9  years,  SD  =  1 1.6  years,  range  =  24-58  years.  All 
subjects  had  completed  some  college  course  work.  Two  subjects  were  TCC  qualified. 

Materials 


Background/Computer  Experience  Survey  .  This  paper-and-pencil  instrument 
was  composed  of  items  tapping  selected  background  variables  including:  age,  sex,  rank 
military  occupational  specialty  (MOS),  years  in  service,  and  educational  level,  as  well  as 
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information  related  to  the  subject’s  prior  experience  with  TCC  and  computers  (see 
Appendix  B  for  all  questionnaires). 


TCC  Principles  of  War  Module  Evaluation  Questionnaire.  Two  versions  of  this 
paper-and-pencil  instnunent  were  constructed.  Subjects  in  the  control  condition 
completed  a  three  item  survey  which  included  items  tapping  the  subject’s  exposure  to  the 
concepts  on  the  end-of-module  quiz,  difficulty  of  quiz  items,  and  comprehension  of  the 
items. 


Subjects  in  the  experimental  condition  completed  a  more  detailed  version  of  the 
control  questionnaire  which  tapped  the  interest  level  of  the  module,  usefulness  of 
information  presented  in  the  module,  usefulness  of  specific  lesson  sections,  i.e.,  learning 
objective,  definitions,  doctrinal  information,  examples,  and  review;  relevance  of 
information  in  the  module  and  of  specific  lesson  sections;  effectiveness  of  examples, 
usefulness  of  the  self-assessments;  difficulty  and  fairness  of  quiz  items,  and  overall 
instructional  value  of  the  module. 

Principles  of  War  Computer-Based  Instructional  Module.  The  Principles  of  War 
module  consisted  of  nine  lessons.  Each  lesson  covered  a  specific  principle  of  war: 
objective,  offensive,  mass,  economy  of  force,  maneuver,  unity  of  command,  security, 
surprise,  and  simplicity.  For  continuity  purposes  and  programming  ease,  each  lesson  was 
organized  in  an  identical  format.  Lessons  started  with  a  screen  detailing  the  learning 
objective.  This  was  followed  by  sections  which  focused  on:  definitions  (of  the  specific 
principle  of  war),  doctrinal  information,  historical  examples  (Battle  of  the  Bulge),  react  to 
contact  drills,  lesson  review,  and  self-assessment  questions.  (See  Figures  1-5  for  screen 
design  examples.)  Each  lesson  requires  approximately  seven  to  ten  minutes  to  complete. 
Once  the  subject  completes  all  nine  lessons,  he  or  she  is  then  presented,  in  this  prototype 
module,  with  a  27-item  multiple-choice  quiz.  Due  to  time  constraints  on  the  part  of  the 
course  developers,  the  end-of-module  quiz  items  were  identical  to  the  self-assessment 
items  used  at  the  end  of  each  of  the  nine  lessons,  but  presented  in  a  different  order.  The 
quiz  requires  approximately  15-20  minutes  to  complete. 
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Figure  2.  TCC  Principles  of  War  Learning  Objective  Screen. 
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Figure  3.  TCC  Principles  of  War  Battle  of  the  Bulge  Screen. 


Figure  4.  TCC  Principles  of  War  React  to  Contact  Screen. 
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Figure  5.  TCC  Principles  of  War  “Objective  Dog”  Screen. 

Design  and  Procedure 

The  study  employed  a  randomized  two-group,  post-test  only  design.  The 
independent  variable  was  instruction  (CBI  vs  No  CBI). 

Subjects  arrived  at  the  Multimedia  Training  Development  Division  office  in  pairs. 
Prior  to  their  arrival,  each  pair  of  subjects  was  randomly  assigned  to  either  the 
experimental  (CBI)  or  control  (No  CBI)  group.  The  researchers  introduced  themselves 
and  explained  the  purpose  of  the  evaluation.  Subjects  were  informed  of  their  role  in  the 
evaluation,  what  was  expected  of  them,  and  completed  the  Background/Computer 
Experience  Survey. 

Subjects  in  the  experimental  condition  were  told  that  they  would  go  through  the 
entire  Principles  of  War  module,  answer  all  self-assessment  items  following  each  lesson, 
and  take  the  27-item  multiple-choice  quiz  at  the  end  of  the  module.  Subjects  were  also 
told  that  they  would  work  by  themselves  at  their  own  work  station  in  separate  rooms.  All 
instruction  (CBI/CD-ROM)  was  presented  on  486-compatible  personal  computers.  The 
module  was  loaded  onto  each  computer  prior  to  the  arrival  of  the  subjects.  Each  subject 
began  the  Principles  of  War  module  on  the  introduction  screen.  One  researcher  was 
assigned  to  each  work  area  to  observe  the  subject  and  note  any  problems  that  the  subject 
might  have  going  through  the  module.  The  researchers  tried  to  be  as  unobtrusive  as 
possible.  Subjects  were  encouraged  to  report  their  perceptions  of  the  module  to  the 
researcher  as  they  progressed  through  each  of  the  nine  lessons.  They  could  requesx 
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assistance  from  the  researcher  if  they  had  difficulty  with  the  navigation  tools.  However, 
questions  related  to  the  lesson  content  or  quiz  items  were  discouraged. 

The  researchers  emphasized  to  the  subjects  the  importance  of  proceeding  through 
the  lessons  carefully  and  doing  their  best  on  the  self-assessment  questions  and  the  27- 
item  multiple-choice  quiz  at  the  end  of  the  module.  Observations  from  earlier  piloting 
suggested  that  most  subjects  completed  the  Principles  of  War  module  in  about  one-and-a- 
half  to  two  hours.  Based  on  these  observations,  a  three  hour  time  block  was  allotted  to 
ensure  completion  of  the  lessons  and  the  end-of-module  quiz  by  all  subjects.  Upon 
completion  of  the  module  and  the  quiz,  subjects  were  debriefed  and  completed  the 
experimental  version  of  the  TCC  Principles  of  War  Module  Evaluation  Questionnaire. 

For  the  control  condition,  subjects  were  instructed  that  they  would  take  the  end- 
of-module  quiz  only.  Control  subjects  were  also  assigned  to  their  own  computers  and 
work  stations,  and  were  observed  by  one  of  the  researchers.  Like  the  experimental 
subjects,  the  control  subjects  began  with  the  introduction  screen  and  proceeded  to  the 
menu.  From  the  menu,  they  were  instructed  to  select  the  end-of-module  quiz.  Control 
subjects  were  allotted  one  hour  to  complete  the  quiz.  Upon  completion  of  the  quiz, 
subjects  were  debriefed  and  then  completed  the  control  version  of  the  TCC  Principles  of 
War  Module  Evaluation  Questionnaire. 


Results 


Computer  Experience 

Analyses  conducted  on  the  Background/Computer  Survey  indicated  that  the 
subjects  were  quite  familiar  with  computers.  Ninety-four  percent  of  the  subjects  had 
some  experience  working  with  computers.  On  the  average,  subjects  had  been  working 
with  computers  for  approximately  three  years  (35  months),  range  =  0-11  years.  Sixty- 
nine  percent  of  the  subjects  had  some  formal  computer  training.  The  majority  of  the 
subjects  (81 .3%)  currently  use  computers  in  their  jobs  as  part  of  their  duties.  Subjects,  to 
a  large  extent  (81.3%),  felt  comfortable  using  computers.  However,  only  thirty-seven 
percent  (37.5%)  of  the  subjects  said  that  they  owned  a  personal  computer. 

End-of-Module  Quiz  Performance 

Means  and  standard  deviations  were  calculated  for  the  quiz  scores  for  subjects  by 
condition.  The  experimental  group  (n  =  8)  answered  an  average  of  23.9  items  correctly 
out  of  27  total  items  (88.9%),  SD  =  2.10,  range  20  -  26.  In  contrast,  the  control  group 
(n  =  8)  answered  an  average  of  13.1  out  of  27  items  correctly  (48.1%),  SD  =  4.88,  range 
=  8-21.  This  difference  was  statistically  significant,  F  (1,  14)  =  32.73,  p  =  .0001. 
Despite  the  small  sample  size,  these  findings  (i.e.,  the  magnitude  of  the  group  difference 
and  the  significant  F  value)  suggest  that  the  group  difference  was,  in  fact,  reliable. 
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The  group  difference  may  be  somewhat  inflated,  however,  and  should  be 
interpreted  cautiously.  As  was  noted  earlier,  due  to  time  constraints  on  the  part  of  the 
course  designer,  the  end  of  module  quiz  items  were  identical  to  the  self-assessment  items 
received  at  the  end  of  each  lesson.  The  impact  that  this  item  exposure  may  have  had  on 
subject  performance  is  unclear.  Although  they  were  not  formally  questioned,  based  on 
the  observations  and  remarks  made  during  the  end-of-module  quiz,  the  subjects  did  not 
seem  to  be  aware  of  the  connection  between  the  self-assessment  and  quiz  items. 

Pearson  product-moment  correlations  were  computed  between  quiz  scores  and 
computer  background,  age,  status  (military  vs  civilian),  TCC  qualification,  and  treatment 
condition.  The  results  showed  only  one  correlation  reached  statistical  significance.  High 
quiz  scores  were  significantly  correlated  with  subjects  having  received  the  module  of 
instruction,  r(16)  =  -.84,  p  =  .0001.  Computer  background,  experience,  level  of  comfort 
working  with  computers,  etc.,  were  not  significantly  correlated  with  quiz  performance. 
Additional  analyses  (Chi-square)  revealed  no  significant  group  differences  for 
educational  level. 

Further  analysis  of  the  data  showed  that  the  two  subjects  reporting  prior  TCC 
qualification  were  assigned  to  the  experimental  group.  However,  their  quiz  scores  were 
not  significantly  higher  than  the  scores  obtained  by  the  remaining  subjects  assigned  to  the 
group.  It  appears,  therefore,  that  prior  TCC  qualification  did  not  artificially  inflate  the 
experimental  group  scores. 

Subjective  Evaluations  of  the  Principles  of  War  Module 

Experimental  subjects  were  asked  to  rate  the  interest  level,  overall  usefulness  and 
relevance  of  the  module.  Seventy-one  percent  (71.5%)  of  the  experimental  subjects  felt 
that  the  content  of  the  module  was  very  or  somewhat  interesting. 

Usefulness  of  specific  lesson  sections.  With  regard  to  usefulness,  all  respondents 
thought  that  the  information  presented  in  the  module  was  either  very  useful  (37.5%)  or 
somewhat  useful  (62.5%).  Each  lesson  section  was  also  rated  for  usefulness.  Table  1 
shows  the  breakdown  of  ratings  by  section. 
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Table  1 


Usefulness  Ratings  for  the  Principles  of  War  Module  by  Lesson  Section 
(in  percent) 


Very 

Useful 

Somewhat 

Useful 

Somewhat 

Useless 

Very 

Useless 

Learning 

Objective 

37.5" 

12.5 

12.5 

37.5 

Definition 

62.5 

25.0 

0.0 

12.5 

Doctrinal 

Information 

62.5 

12.5 

12.5 

12.5 

Battle  of  the 
Bulge 

25.0 

12.5 

25.0 

37.5 

React  to 
Contact 

12.5 

25.0 

50.0 

12.5 

Review 

12.5 

0.0 

0.0 

87.5 

“  n  =  8  for  all  cells. 

As  can  be  seen  from  Table  1,  sections  devoted  to  defining  and  explaining  the 

principle  (doctrinal  information)  were  seen  as  the  most  useful  sections.  The  review 
section  was  seen  by  virtually  all  subjects  (87.5%)  as  useless.  This  was  due  in  large  part 
to  the  fact  that  the  review  section  did  not  summarize  key  points,  but  simply  instructed 
students  to  go  back  to  the  appropriate  section(s)  in  the  lesson  if  they  felt  they  needed  to 
review  an  area.  For  the  learning  objective  and  to  a  lesser  extent  the  Battle  of  the  Bulge 
and  react  to  contact  sections,  subjects  seemed  somewhat  split  in  their  reactions  of  the 
usefulness  of  these  sections.  For  the  learning  objective  section,  one  half  of  the  subjects 
thought  that  it  was  very  or  somewhat  useful  and  the  other  half  thought  that  it  was 
somewhat  or  very  useless.  For  the  Battle  of  the  Bulge  and  react  to  contact  sections,  sixty- 
two  percent  of  the  subjects  felt  that  these  sections  were  either  somewhat  or  very  useless. 

Relevance  of  specific  lesson  sections.  Sixty-two  percent  (62.5%)  of  the 
experimental  subjects  felt  that  the  information  presented  in  the  module  seemed  mostly 
relevant  to  the  stated  instructional  goals.  Table  2  summarizes  the  relevance  ratings  given 
by  the  subjects  for  each  lesson  section. 
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Table  2 


Relevance  Ratings  for  the  Principles  of  War  Module  by  Lesson  Section 
(in  percent) 


Very 

Relevant 

Somewhat 

Relevant 

Somewhat 

Irrelevant 

Ver>' 

Irrelevant 

Learning 

Objective 

37.5® 

12.5 

12.5 

37.5 

Definition 

75.0 

12.5 

12.5 

0.0 

Doctrinal 

Information 

75.0 

25.0 

0.0 

0.0 

Battle  of  the 

Bulge 

12.5 

37.5 

12.5 

37.5 

React  to 

Contact 

37.5 

37.5 

12.5 

12.5 

Review 

12.5 

0.0 

0.0 

87.5 

*  n  =  8  for  all  cells. 


As  was  the  case  for  the  ratings  of  usefulness,  sections  devoted  to  defining  and 
explaining  the  principle  (doctrinal  information)  were  seen  as  the  most  relevant,  in  terms 
of  providing  information  supporting  the  stated  instructional  goals.  The  learning  objective 
and  the  Battle  of  the  Bulge  sections  seemed  to  draw  mixed  reactions  from  the  subjects. 
One  half  of  the  subjects  thought  that  these  sections  provided  very  or  somewhat  relevant 
information.  The  other  half  thought  that  these  sections  provided  somewhat  or  very 
irrelevant  information.  For  the  react  to  contact  section,  seventy-five  percent  of  the 
subjects  felt  that  this  section  presented  information  that  was  very  or  somewhat  relevant  to 
the  stated  instructional  goals.  Finally,  the  review  section  was  seen  as  very  irrelevant  by 
virtually  all  respondents  (87.5%). 

Effectiveness,  test  difficulty,  and  instructional  value  ratings.  The  majority  of  the 
experimental  subjects  felt  that  the  Battle  of  the  Bulge  example  (62.5%)  and  the  react  to 
contact  drills  (75%)  were  very  or  somewhat  effective  in  helping  them  to  understand  the 
principles  of  war.  All  subjects  thought  that  the  self-assessment  questions  at  the  end  of 
each  lesson  were  either  very  or  somewhat  useful  in  helping  them  learn  the  principles  of 
war. 
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With  regard  to  the  quiz  given  at  the  end  of  the  module,  thirty-seven  percent 
(37.5%)  of  the  subjects  thought  the  items  were  somewhat  difficult  and  fifty  percent  felt 
that  they  were  somewhat  easy.  Eighty-seven  percent  (87.5%)  of  the  experimental 
subjects  felt  that  the  quiz  adequately  assessed  their  knowledge  of  the  principles  of  war. 
Overall  ratings  of  the  instructional  value  of  the  Principles  of  War  module  were  positive. 
The  percentage  breakdown  was  as  follows:  Excellent-37.5%,  Good-12. 5%,  Fair-37. 5%, 
Poor-12.5%. 

Additional  Comments  on  Lesson  Structure  and  Content 


Comments  made  by  the  subjects  as  they  went  through  the  module  and  their  open- 
ended  responses  on  the  TCC  Evaluation  Questionnaire  were  collected  and  summarized. 
Comments  were  categorized  by  lesson  section  or  topic  area  and  are  presented  in  the 
following  sections. 

Learning  objective.  The  learning  objective  section  (screen)  was  seen  as  too 
generic.  The  current  approach,  which  involves  the  presentation  of  a  generic  screen  for 
each  lesson  referring  to  “this  principle  of  war”  was  not  helpful  to  the  subjects,  who 
frequently  had  to  ask  what  lesson  (principle  of  war)  they  were  in  at  the  time. 

Definition.  The  definition  should  not  incorporate  the  word  or  principle  that  is  to 
be  defined,  e.g.,  FM  100-5  defines  mass  as  “Mass  the  effects  of  overwhelming  combat 
power  at  the  decisive  place  and  time.”  This  was  noted  by  several  subjects  for  certain 
principles  of  war  such  as  mass  and  objective. 

Doctrinal  information.  The  presentation  of  doctrinal  information  was  also 
criticized  by  some  subjects.  They  thought  that  the  language  and  terms  used  for  some  of 
the  information  was  too  complex,  unfamiliar,  or  too  complicated. 

Battle  of  the  Bulge.  Examples  were  seen  by  the  subjects  as  too  similar  to  each 
other.  Subjects  did  not  know  which  principle  was  being  illustrated  at  times.  Examples 
for  some  principles  of  war  did  not  clearly  demonstrate  the  principle  in  question.  The 
subjects  felt  that  more  diversified  samples,  specific  to  the  principle  of  war  selected,  were 
needed.  Subjects  indicated  that  they  were  tired  of  seeing  what  they  felt  were  basically  the 
same  examples  repeated  over  and  over. 

React  to  contact.  The  scenario  used  for  the  react  to  contact  drill  was  the  same  for 
each  principle  of  war.  In  the  present  format,  key  aspects  of  the  principle  of  war  selected 
could  be  gleaned  by  the  student  only  from  the  post-scenario  narration  which  attempted  to 
connect  the  scenario  to  the  principle. 

Review.  Almost  every  subject  skipped  the  review  section  on  every  lesson  after 
the  first  or  second  exposure.  Subjects  thought  the  review  would  be  more  of  a  summary 
(e.g.,  a  section  providing  bullet  summary  points).  Its  present  format,  telling  students  to 
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go  back  and  review  selected  sections  in  the  lesson,  was  not  seen  by  the  subjects  as  ver>' 
useful  or  relevant. 

Self-assessment  items.  Although  most  subjects  felt  the  self-assessment  section 
was  useful  and  relevant,  they  did  not  like  the  questions  whose  stems  included  the  phrase 
“Which  of  the  following  is  not...”  Better  distracter  items  are  needed,  preferably  those  that 
are  related  to  the  principle  of  war  being  studied  instead  of  using  definitions  from  other 
principles  of  war.  For  many  items  the  distracters  are  so  obviously  unrelated  to  the 
principle  being  assessed  that  the  subject  could  easily  eliminate  one  or  two  alternatives. 

Final  exam.  The  final  exam  was  problematical  in  that  it  was  composed  of  the 
self-assessment  items  from  each  of  the  lessons  but  presented  to  the  subject  in  a  different 
order.  The  present  format  potentially  confounds  the  treatment  effect  with  any  incidental 
learning  that  may  have  occurred  from  previous  exposure  to  the  items  during  the  self- 
assessment  portions  of  the  lessons. 

Audio.  The  volume  was  not  consistent  throughout  the  lessons.  Certain  sections 
were  extremely  loud  and  many  subjects  had  to  turn  the  volume  down  on  these  sections 
and  turn  it  up  for  the  next  section  (s).  The  fluctuations  proved  to  be  very  distracting  for 
the  subjects. 

Navigation.  Some  subjects  had  problems  navigating  in  the  program,  e.g.,  getting 
disoriented  at  the  end  of  the  lesson,  having  trouble  returning  to  the  main  directory,  or 
difficulty  exiting  a  lesson  and  entering  the  next  lesson. 

Motivation.  Some  subjects  felt  that  the  instruction  was  boring,  particularly  the 
Battle  of  the  Bulge  and  react  to  contact  sections  (though  not  necessarily  useless, 
irrelevant,  or  ineffective).  This  was  primarily  due  to  the  similarity  among  many  of  the 
Battle  of  the  Bulge  examples  used  in  demonstrating  key  aspects  of  the  principle  of  war  in 
question.  The  react  to  contact  drill  scenarios  were,  for  all  intents  and  purposes,  identical 
across  lessons. 


Discussion 

The  results  of  this  evaluation  suggest  that  the  prototype  CBI  Principles  of  War 
module  was  effective  in  teaching  doctrinal  material,  specifically  the  definitions  of  the 
nine  principles  of  war  and  supporting  doctrinal  information.  As  noted  earlier,  the 
difference  between  the  groups  on  the  final  quiz  may  be  somewhat  inflated  since  the  items 
used  for  the  quiz  were  the  same  as  those  used  for  the  self-assessment  questions  for  each 
lesson.  Some  subjects  indicated  that  certain  questions  seemed  familiar,  but  this  may  be 
due  to  the  similarity  of  item  stems,  which  were  virtually  identical  for  certain  sets  of 
questions,  e.g.,  definition,  doctrinal  information.  In  no  instance  did  the  researchers  get 
any  sense  that  the  subjects  recognized  both  the  question  and  the  correct  response  for  that 
question.  This  issue  has  been  noted  to  the  course  designers  and  will  be  corrected  for  the 
final  principles  of  war  module. 
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Subjective  Value  of  the  Lesson  Sections 


Extensive  feedback  was  solicited  from  the  subjects  on  the  organization  of  the 
module  and  the  value  of  the  subject  content.  The  feedback  indicated  that  certain 
sections  clearly  need  to  be  modified.  The  review  section  was  seen  by  the  subjects  as 
virtually  worthless.  The  learning  objective  screen,  although  viewed  by  half  of  the 
subjects  as  very  or  somewhat  useful,  can  also  be  improved.  In  both  instances, 
modifications  can  be  made  rather  easily.  The  review  section  must  include  key  points 
summarizing  major  issues  presented  in  the  lesson.  The  learning  objective  section  can  be 
presented  at  the  beginning  of  the  module  as  part  of  the  introduction  to  the  instruction 
which  all  students  are  exposed  to  when  they  log  on  for  the  first  time.  Since  all  lessons 
have  the  same  action,  conditions,  and  standards,  there  is  no  need  to  repeat  the  same 
generic  screen  (section)  for  each  lesson. 

The  Battle  of  the  Bulge  and  react  to  contact  sections,  while  appearing  to  have 
some  relevance  and  utility  for  learning  the  course  material,  could  be  greatly  improved, 
particularly  the  react  to  contact  drill.  The  drill  scenarios  that  were  presented  were  seen  by 
the  subjects  as  boring.  This  is  understandable  since  the  geography,  mission,  enemy,  and 
situation  were  always  the  same.  Using  different  geographic  settings  and  mission 
scenarios,  manipulating  the  enemy  composition  and  situation,  and  tailoring  the  scenarios 
for  a  specific  principle  of  war  would  greatly  enhance  the  value  of  this  section. 

Frequently,  the  subjects  had  little  idea  what  principle  of  war  they  were  viewing  since  the 
scenarios  were  always  the  same.  For  many  of  the  scenarios,  it  was  only  when  the  subject 
heard  the  post-scenario  narration  that  he  realized  what  principle  of  war  he  was  viewing. 

The  Battle  of  the  Bulge  section  attempted  to  provide  different  examples  to  try  to 
match  the  principle  of  war  that  was  being  taught.  However,  subjects,  at  times,  still  did 
not  grasp  the  connection  between  the  example  and  the  principle  or  felt  that  some  of  the 
examples  were  too  similar  to  adequately  allow  the  them  to  distinguish  the  key  aspects  of 
the  principle  of  war  in  question. 

For  both  the  Battle  of  the  Bulge  and  react  to  contact  sections  major  changes 
would  be  required.  The  work  required  to  correct  these  shortcomings  would  be  extensive 
and  current  course  production  deadlines  make  it  unlikely  that  major  changes  will  be  made 
in  these  sections. 

The  feedback  received  on  the  definition  and  doctrinal  information  sections 
seemed  to  be,  for  the  most  part,  very  useful  and  relevant.  Although  the  phrasing  of  terms 
and  information  was  seen  by  a  few  subjects  as  too  complex  or  awkw'ard,  course 
developers  indicate  that  no  changes  can  be  made  since  the  definitions  must  be  exactly 
those  used  in  the  FM, 

With  regard  to  the  self-assessment  and  quiz  items,  several  modifications  are 
required.  First,  all  self-assessment  items  whose  stem  includes  the  phrase  “Which  of  the 
following  is  not...”  should  be  changed  to  a  positive  statement,  e.g.,  “Which  of  the 
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following  is  an  example  of...”  Subjects  found  these  negative  items  very  difficult  to 
“retranslate”  or  process.  In  many  instances,  the  retranslation  process  clearly  slowed  the 
subjects’  response  time.  At  times  this  lead  them  to  respond  with  the  UTong  answer. 

Tamir  (1993)  found  that  while  both  modes  of  questions  (i.e.,  positive  and  negative 
phrasing)  are  valid,  significant  performance  differences  were  found  for  items  of  high 
cognitive  value,  i.e.,  items  involving  interpretation,  or  drawing  conclusions.  In  Tamir’s 
study  smdents  performed  significantly  better  on  the  more  cognitively  challenging 
questions  that  were  written  in  the  positive  mode  than  those  that  were  written  in  the 
negative  mode.  For  less  difficult  items,  differences  between  positive  and  negative  modes 
of  questions  were  negligible. 

Maixelli  (1995)  stresses  the  use  of  the  positive  rather  than  the  negative  form  in  the 
stem  whenever  possible.  If  a  negative  form  of  the  stem  is  required,  she  recommends 
capitalizing  or  underlying  the  negative  term. 

As  implied  earlier,  the  final  version  of  the  principles  of  war  module  will  have  quiz 
questions  that  will  be  different  in  structure  (wording)  and  format  from  the  self-assessment 
items.  The  screening  quiz  and  the  final  quiz  for  those  who  opt  not  to  test  out  of  the 
module  will  be  identical  -  matching  definitions,  doctrinal  information  and  examples  to  the 
nine  principles  of  war.  The  self-assessment  items  will  be  multiple-choice.  For  those 
students  who  test  out  of  only  portions  of  the  module  or  for  those  opting  to  start 
immediately  with  the  lessons  but  fail  specific  sections  of  the  final  exam,  a  different 
testing  format  will  be  used  for  retesting  these  individuals.  Specifically,  after  the  students 
repeat  the  specific  lesson(s)  they  will  be  retested  on  the  specific  principle(s)  of  war 
covered  by  the  lesson(s)  using  a  multiple-choice  format. 

Conclusion 

The  Principles  of  War  module  represents  only  one  of  five  modules  that  will  be 
developed  for  the  TCC.  Each  of  these  modules  will  go  through  the  same  (or  a  similar) 
evaluative  process  as  the  Principles  of  War  module  described  in  this  report.  The  results 
of  this  evaluation  indicated  that  the  Principles  of  War  module,  as  constructed,  is  effective 
at  teaching  knowledge  about  selected  concepts.  Sections  of  the  module  could  be 
significantly  improved,  however. 

Survey  and  observational  data  indicated  that  certain  sections  such  as  The  Battle  of 
the  Bulge  and  React  to  Contact  drills,  while  potentially  very  useful,  would  require 
extensive  modifications.  Other  sections  (Definition,  Doctrinal  Information)  of  the 
Principles  of  War  module  were  useful  in  promoting  learning  and/or  provided  relevant 
information  in  support  of  the  stated  instructional  goals.  These  sections  require  relatively 
minor,  if  any,  refinements.  The  Learning  Objective  section  or  screen  can  be  easily 
improved.  The  Review  section,  while  seen  as  virtually  worthless  in  its  present  format, 
can  be  greatly  improved.  The  modifications  required  in  this  section  can  be  made  quite 
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easily.  The  Self-Assessment  section,  despite  the  negative  wording  of  some  questions, 
was  nevertheless  seen  as  useful  to  the  students  in  reinforcing  key  points  in  the  lessons. 

The  results  of  the  formative  evaluation  process  described  in  this  report  will  help 
ensure  that  the  distance  learning  technologies  selected  for  the  TCC  are  optimally 
employed. 
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Appendix  A 


Personalized  System  of  Instruction  (PSI): 

Overview  and  Empirical  Documentation 

PSI:  An  overview.  According  to  Keller,  who  originated  the  PSI  method,  (cited  in 
Buskist,  Cush,  &  DeGrandpre,  1991),  PSI  has  five  essential  characteristics:  self-pacing, 
unit  mastery,  lectures  and  demonstrations,  written  communication,  and  the  use  of 
proctors.  The  self-pacing  feature  enables  students  to  take  quizzes  in  sequence,  at  a  pace 
set  by  the  students  themselves.  This  feature  is  included  to  accommodate  students  and 
adults  with  hectic  schedules  and  work  priorities.  It  also  allows  students  to  proceed  at 
their  own  individual  rates  based  on  their  own  unique  study  habits  and  differences  in 
academic  ability.  Finally,  the  self-pace  feature  gives  the  student  the  opportunity  to 
complete  a  course  ahead  of  the  scheduled  completion  date,  which  in  many  instances  can 
serve  as  a  powerful  reinforcer  for  students,  leaving  more  time  for  other  courses  and 
obligations. 

Unit  mastery.  Students  must  master  each  unit  of  study  before  they  are  permitted 
to  proceed  to  the  next  unit.  The  criterion  for  mastery  varies,  depending  on  the  course 
content  and  objectives.  Criterion  ranges  have  been  reported  as  low  as  80  and  as  high  as 
95  percent  correct.  Units  are  small  parcels  of  information,  typically  short  assignments 
based  around  a  few  central  points.  Unit  quizzes  are  generally  multiple-choice,  fill-in-the 
blank,  and  short  answer.  To  help  prepare  students  for  quizzing,  study  questions  which 
are  aimed  at  flushing  out  key  concepts  are  inserted  in  the  unit.  An  important  component 
of  this  principle  is  that  students  are  permitted  to  take  and  retake,  without  penalty,  unit 
quizzes  until  they  master  the  material  covered  in  the  unit.  If  students  miss  only  one  or 
two  questions  on  a  quiz,  they  may  defend  or  clarify  their  responses  to  the  proctor.  If 
students  show  verbally  that  they  do  indeed  understand  the  material,  they  are  considered  to 
have  mastered  it,  and  may  move  on  to  the  next  unit. 

Lectures  and  demonstrations.  In  the  PSI  format,  the  lecture  is  viewed  primarily  as 
a  motivational  tool,  to  be  used  mainly  for  its  power  to  inspire  students  to  study  further. 
Lectures  included  in  PSI  courses  are  typically  short,  perhaps  30  minutes,  and  occur  only 
about  once  a  week.  Attendance  is  not  required  as  subsequent  quizzes  or  tests  do  not 
include  material  from  it.  The  most  important  function  of  the  lectures  is  to  clarify  or 
synthesize  textual  information  and  to  answer  student  questions. 

Written  communication.  Almost  all  communication  between  a  student  and  the 
instructor  or  proctor  is  written.  Students  devote  most  of  their  time  to  reading  or  viewing 
materials  provided  by  the  instructor.  Quizzes  cover  only  these  materials  and  require  that 
the  student  respond  by  writing  a  short  answer  or  choosing  a  multiple-choice  item.  The 
key  to  this  approach  is  the  requirement  that  the  smdent  actively  read,  study,  and  respond 
to  questions  over  unit  materials.  To  insure  students  are  able  to  discriminate  important 
material  from  the  unimportant,  detailed  behavioral  objectives  or  study  questions 
(discussed  earlier)  are  often  incorporated  in  the  unit. 
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Proctors.  Proctors  are  selected  from  a  pool  of  students  who  have  taken  a  previous 
section  of  the  PSI  course  and  have  met  several  other  criteria.  The  proctor  is  typically 
chosen  for  his  demonstrated  mastery  of  the  course  content,  for  his  understanding  of  the 
special  problems  that  confront  the  student  as  a  beginner,  and  for  his  willingness  to  assist. 
The  use  of  proctors  provides  the  personal  or  social  aspect  usually  lost  by  having  fewer 
lectures.  The  proctor,  as  a  peer  who  is  intimately  aware  of  the  nuances  of  preparing  for 
and  mastering  unit  quizzes,  is  seen  as  someone  who  can  relate  more  effectively  to  the 
student  than  the  instructor,  providing  a  personal  touch  to  the  learning  process. 

PSI  Effectiveness:  Outcome  Research 


Buskist,  Cush,  and  DeGrandpre  (1991)  reviewed  a  number  of  studies  which  have 
investigated  or  summarized  the  results  of  research  aimed  at  comparing  the  relative 
effectiveness  of  PSI  with  other  pedagogical  methods  in  facilitating  knowledge  acquisition 
and  enhancing  overall  academic  performance.  In  general,  specific  studies  have  shown 
that,  compared  to  traditional  lecture  methods  or  other  appropriate  controls  using  the  same 
reading  assignments,  students  in  PSI  courses  have  done  better  on:  1)  multiple-choice 
examinations  (McMichael  &  Corey,  1969);  2)  final  examinations  (Corey  &  McMichael, 
1970);  3)  essay  examinations  (Sheppard  &  MacDermot,  1970);  and  4)  examinations 
designed  specifically  to  test  recall  and  application  (Morris  &  Kimbrell,  1 972).  Even  with 
the  number  of  test  questions  controlled  or  partially  controlled  for,  PSI  produced  better 
student  performance. 

Extensive  reviews  comparing  PSI  to  traditional  methods  of  instruction  clearly 
show  the  superiority  of  PSI.  Taveggia  (1976)  analyzed  the  results  of  fourteen  studies 
which  looked  at  course  examinations  across  a  variety  of  content  areas  including 
introductory  psychology,  learning,  cultural  anthropology,  chemistry,  and  electrical, 
mechanical,  and  nuclear  engineering.  He  found,  without  exception,  that  students  did 
better  under  PSI  methods  than  under  traditional  ones. 

In  another  review  of  31  studies  by  Kulik  (1976)  comparing  PSI  to  other  teaching 
methods  across  a  wide  range  of  subject  areas,  81  percent  of  the  cases  showed  that  the 
final  exam  performance  for  PSI  students  was  statistically  superior  to  those  in  traditional 
courses.  Kulik  also  found  that  retention  and  transfer  were  superior  using  PSI  compared 
to  traditional  methods  of  teaching. 

Kulik,  Kulik,  and  Cohen  (1979)  conducted  a  meta  analysis  of  75  comparative 
studies  pitting  PSI  against  traditional  methods  of  instruction.  Among  the  many  findings 
that  were  reported,  it  was  found  that  PSI  examination  scores  were  significantly  higher 
(by  about  eight  points)  than  scores  obtained  from  lecture  based  courses.  Final  course 
grades  were  also  significantly  higher  in  the  PSI  courses.  Using  the  traditional  four  point 
scale,  with  4  being  superior  and  1  passing,  PSI  courses  averaged  3.09.  In  comparison, 
the  control  classes  averaged  2.3 1 .  Kulik,  Kulik,  and  Cohen  (1979)  reported  that  the  PSI 
programs  they  studied  raised  final  examination  scores  by  about  .50  standard  deviations 
over  programs  using  conventional  (non-PSI)  means  of  instruction.  They  also  found  that 
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PSI  produced  less  variation  in  achievement,  higher  student  ratings,  and  fewer  course 
withdrawals  and  that  these  favorable  results  occurred  across  a  variety  content  areas  and 
course  settings. 

A  recent  review  by  Fletcher  ( 1 992)  of  individualized  systems  of  instruction 
showed  PSI  to  be  one  of  the  most  effective  systems  employed  based  on  the  magnitude  of 
improvement  in  student  achievement  including  final  exams  and  end-of-course  grades. 

PSI  was  more  effective  than  programmed  instruction  approaches,  audio  tutorial, 
individual  guided  education  and  such  computer-oriented  approaches  as  the  Individually 
Prescribed  Instruction/Adaptive  Learning  Environments  Model  (ALEM),  Program  for 
Learning  in  Accordance  with  Needs  (PLAN),  optimized  instruction,  and  intelligent 
computer-assisted  instruction.  Only,  the  strands  approach  (a  C AI  application  which 
allows  for  individualization  of  pace,  content,  sequence,  and  style  in  the  instruction  of 
basic  component  skills)  resulted  in  more  effective  learning.  However,  the  data  are  sparse. 
In  one  study  mentioned.  Flatter  and  Atkinson  (1973)  reported  an  improvement  of  .81 
standard  deviations  in  achievement  for  a  strands  application  in  beginning  reading.  In 
another  study,  Ragosta,  Holland,  &  Jamison  (1982)  reponed  an  average  improvement  of 
.26  standard  deviations  in  achievement  for  strands  applications  in  mathematics. 

PSI:  Component  Analysis 

In  addition  to  assessing  the  effectiveness  of  the  overall  PSI  model,  researchers 
have  also  looked  at  the  relative  contributions  of  the  five  PSI  components  in  the  learning 
process.  Buskist,  Cush,  and  DeGrandpre  (1991)  reviewed  the  research  in  this  area  and 
found  that  some  general  guidelines  could  be  offered  with  regard  to  PSI  component 
effectiveness.  First,  courses  involving  the  mastery  requirement,  immediate  performance 
feedback,  and  review  units  reliably  produce  high  quality  student  performances.  Second, 
optional  lectures,  self-pacing,  and  the  use  of  student  proctors  as  peer  tutors,  do  not,  in 
themselves,  seem  vital  to  student  success  in  PSI-based  courses.  Finally,  though  self¬ 
pacing  may  not  be  a  critical  feature  of  PSI-based  courses,  students  do  seem  to  appreciate 
it.  The  key  is  to  ensure  students  manage  their  course  time  wisely. 
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Appendix  B 

Background/Computer  Experience  Survey 

In  order  to  help  us  evaluate  this  course  please  provide  some  information  about  yourself  and 
your  experience  with  computers.  All  responses  will  be  kept  anonymous. 

Please  Print 

Id  #  (Last  four  digits  of  SSN): _ 

Age: _ 

Sex: _ Male  _ Female 

Rank: _ 

Component: _ Active  Duty  _ National  Guard/Reserve 

MOS/CMF: _ 

Years  in  service: _ 

Highest  Education  Level  (please  check  one  and  circle  if  necessary): 

_ High  School  diploma  or  GED 

_ Some  College 

_ College  Degree  (please  circle):  Associates  Bachelors 

_ Some  Graduate  School 

_ Graduate  Degree  (please  circle):  Masters  Doctorate 


For  the  following  please  circle  the  response  that  applies  to  you  and  provide  other 
information  when  necessary. 

1.  Have  you  taken  the  Tactics  Certification  Course  (TCC)  before?  Yes  No  Notsure 

If  Yes,  about  when?  Month _  Year _ 

2.  Do  you  have  experience  working  with  computers?  Yes  No 

If  so,  how  long? _ 

3.  Have  you  ever  had  any  training  in  the  use  of  computers?  Yes  No 

If  so,  what  type  of  training?  _ 


4.  Do  you  own  a  personal  computer?  Yes  No 
If  so,  how  frequently  do  you  use  it  and  for  what?. 


5.  Do  you  use  a  personal  computer  as  part  of  your  duties?  Yes  No 

6.  Do  you  feel  comfortable  using  computers?  Yes  No 
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TCC  Principles  of  War  Module 
Evaluation  Questionnaire 

Have  you  had  any  previous  experience  with  the  nine  principles  of  war? 

_ ^Yes 

_ No 

If  yes,  how  much  did  this  exposure  help  you  in  answering  the  quiz  items? 


2.  How  difficult  were  the  quiz  items  for  you? 


Very  Somewhat  Somewhat  Very 

Difficult  Difficult  Easy  Easy 

3.  Were  the  quiz  items  easy  to  understand? 

_ ^Yes 

_ No 

If  no,  explain: 


Please  provide  us  with  any  additional  comments  that  you  feel  may  help  us  in 
evaluating  the  effectiveness  of  this  quiz. 
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TCC  Principles  of  War  Module 
Evaluation  Questionnaire 

1.  How  interesting  was  the  content  of  this  module? 

Very  Somewhat  Somewhat  Very 

Interesting  Interesting  Uninteresting  Uninteresting 

2.  How  useful  to  you  was  the  information  presented  in  this  module? 

Very  Somewhat  Somewhat  Very 

Useful  Useful  Useless  Useless 

How  useful  was  each  of  the  following  sections  for  you?  Rate  each  of  the 
following  sections  from  1  to  4,  with  1  being  very  useful  and  4  being  very 
useless, 

_ Learning  objective 

_ Definitions  (of  specific  principles  of  war) 

_ Doctrinal  information 

_ Battle  of  the  Bulge  examples 

_ React  to  Contact  drills 

_ Review 

3.  How  much  information  presented  in  the  module  seemed  irrelevant  to  the 
instructional  goals  stated  in  the  module? 

Most  information  More  information  More  information  Most  information 

seemed  irrelevant  seemed  irrelevant  seemed  relevant  seemed  relevant 

to  me,  than  relevant  to  me.  than  irrelevant  to  me.  to  me. 

How  much  irrelevant  information  did  each  of  the  following  sections 
contain?  Rate  each  of  the  following  sections  from  1  to  4,  with  1  being  mostly 
irrelevant  information  and  4  being  mostly  relevant  information. 

_ Learning  objective 

_ Definitions  (of  specific  principles  of  war) 

_ Doctrinal  information 

_ Battle  of  the  Bulge  examples 

_ React  to  Contact  Drills 

_ Review 

4.  How  effective  were  the  Battle  of  the  Bulge  examples  in  helping  you  to 
understand  the  principles  of  war? 


Very  Somewhat  Somewhat  Very 

Effective  Effective  Ineffective  Ineffective 
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5. 


How  effective  were  the  React  to  Contact  drills  in  helping  you  to 
understand  the  principles  of  war? 

Very  Somewhat  Somewhat  Ver>' 

Effective  Effective  Ineffective  Ineffective 

6.  How  useful  were  the  self  assessments  (the  short,  end-of-lesson  quizzes)  in 
helping  you  to  learn  the  principles  of  war? 

Very  Somewhat  Somewhat  Very 

Useful  Useful  Useless  Useless 

7.  How  difficult  were  the  items  on  the  final  quiz  for  you? 


Very  Somewhat  Somewhat  Very 

Difficult  Difficult  Easy  Easy 

8.  Do  you  feel  that  the  items  on  the  final  quiz  adequately  assessed  your 
knowledge  of  the  principles  of  war? 

_ ^YES 

_ NO 

9.  Overall,  how  would  you  rate  the  instructional  value  of  the  principles 
of  war  module? 


Excellent  Good  Fair  Poor 


Please  provide  us  with  any  additional  comments  that  you  feel  may  help  us  in 
evaluating  the  effectiveness  of  this  module. 


